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THE HUMAN AS AN OPTIMAL 


CONTROLLER AND INFORMATION PROCESSOR 

by 

Sheldon Baron and David L. Kleinman 


SUMMARY 

A mathematical model of the human operator in multivariable 
control tasks is developed by considering the human as a control 
and information-processing system. The model contains elements 
for describing the operator's inherent physiological limitations 
as well as his instrument-monitoring, data-reconstruction and con- 
trol behavior. Special emphasis is placed on the instrument-moni- 
toring aspects of the model. 

The human's limitations are modelled by combining them into 
an equivalent perceptual time delay and an equivalent observation 
noise. The main assumption underlying the subsequent theoretical 
investigations is that the well-trained, well-motivated operator 
behaves in a near optimal manner, subject to the constraints im- 
posed by the above limitations. Thus, the operator's control be- 
havior is assumed to be that of an ideal feedback controller. 

The human's data-reconstruction process is chosen so as to obtain 
a "best" estimate of the state of the controlled element based on 
information obtained from "sampling" the various instruments. 

The data reconstructor consists of a Kalman estimator and a pre- 
dictor in tandem, its structure is fixed but it depends, para- 
metrically, on the sampling behavior. 

Instrument-monitoring behavior depends explicitly on the con- 
trol task and on the control actions . Provision is made for the 
ability to obtain information from the peripheral visual field and 
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there are no restrictions on signal coupling. The visual sampling 
model also includes means for constraining instrument scanning 
rates. The specific characteristics of the operator’s visual 
sampling behavior are predicted by solving a nonlinear optimiza- 
tion problem. This problem is precisely formulated and methods 
for its solution are discussed. By changing the variances of the 
observational noises it is possible to predict the effects that 
changes in the visual display panel will have upon the human's 
sampling behavior. Finally, instrument sampling characteristics 
for a simple two-axis compensatory tracking task are obtained. 

The results exhibit the general characteristics one would expect 
from a human operator performing a similar task. 
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INTRODUCTION 


The basic objective of research to be performed under 
Contract No. NAS-12-104 is the investigation of representations 
of the human operator with a view towards their relation to dis- 
play system evaluation. Our previous efforts under this contract 
were primarily concerned with developing new methods for identi- 
fying the characteristics of human pilot control behavior. In 
that work we examined linear and nonlinear representations of the 
pilot in a single-axis compensatory tracking task (Ref. 1). Such 
models are useful for obtaining a better understanding of pilot 
control behavior and, hence, for synthesizing manual control sys- 
tems. However, since the models were developed for single-axis 
tracking and required no visual sampling, they represented only a 
first step in the development of analytic display evaluation tech- 
niques . 

We are now investigating the problem of manual control in 
more complex, multivariable situations. Our goal in this phase 
of the research is to obtain a more complete mathematical descrip- 
tion of the human operator's information-processing and control 
behavior. (It should be understood that our aim is not necessarily 
to describe the conscious thought processes or physical structure 
of the operator; rather, we hope to model his overt behavior. Thus 
the models will be valid to the extent that their input-output 
behavior is analogous to that of the human operator.) 

In this report we shall describe our model of the human oper- 
ator as an information-processor and controller. We emphasize 
those aspects of the model concerning visual sampling of multi- 
instrument display panels as this represents the major thrust of 
our recent and current research under this contract. We begin 
with a brief review of past research in human instrument sampling. 
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Then we describe the structure of our model of human operator be- 
havior in a multivariable control task and show how the sampling 
model interfaces with other elements of the overall human operator 
model. The model we have postulated is rooted in modern control 
theory and, as we shall see, to obtain the model parameters (e.g., 
control gains, sampling strategies, etc.), we must solve a complex 
optimization problem. The theoretical background for this problem 
is discussed and we then give a precise mathematical formulation 
of the optimization problem. Numerical results for a simple, two 
axis problem are obtained and these results serve to illustrate 
various aspects of the problem. Finally, we discuss implications 
of our research efforts and the directions for further work. 
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running time 

vehicle state vector 

pilot's control input vector 

external vehicle disturbance vector 

system matrix 

control matrix 

covariance matrix of w 

displayed quantities (C x) 

system output matrix 

human's time delay 

a sampling strategy 

class of all sampling strategies 

observation noise 

covariance matrix of v, , 

—to 

information processed by human operator (£ d (t-x)+ v^(t) 

expectation operator 

terminal or final time 

cost functional to be minimized 

state weighting matrix 

control weighting matrix 

sampling "cost" 

optimal control 

optimal feedback gains 
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SYMBOLS (Continued) 


x least mean squared estimate of x 

— U) — 

J* minimum cost 

w* optimal sampling strategy 

ft range space for function u)(»)eQ 

K time varying solution of matrix Riccati equation 

K asymptotic Riccati Equation 

E "error" covariance matrix 

-to 

I(w) part of J* which depends explicitly on w 
T sampling period 

7 asymptotic variance equation 

T* optimal sampling period 
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VISUAL SAMPLING MODELS: A BRIEF REVIEW 

Although a detailed understanding of human visual sampling 
processes in a complex control or monitoring task is essential to 
the development of rational display design procedures, there has 
been surprisingly little research in this area of human engineer- 
ing. The first significant attempt to understand how pilots use 
their eyes in order to obtain information from many separate in- 
struments was made at Wright-Patterson Air Force Base in the period 
19^9-195^ (Refs. 2-10). The WPAFB studies were aimed at determin- 
ing patterns of pilot eye movements in actual control situations. 

Eye fixations of pilots were recorded in a series of flight tests 
and the frequency and duration of fixations on various instruments, 
as well as transitions between pairs of instruments, were deter- 
mined for a number of flight conditions. However, no attempt was 
made in these studies to model the human visual sampling process. 
Indeed, the inputs to the instruments were not recorded and, con- 
sequently, the data is of limited utility with respect to verify- 
ing any predictive models. 

The first quantitative model for describing pilot sampling 
behavior was developed by Senders (Ref. 11). This model was based 
on information-theoretic ideas, particularly Shannon's sampling 
theorem (Ref. 12). A basic assumption of the model was that the 
human observer samples the various signals periodically and attempts 
to reconstruct the time functions presented on each instrument. 
Moreover, it was assumed that the operator was effectively a single 
channel device capable of attending to only one signal at a time. 
With these concepts as a starting point. Senders was able to derive 
expressions for the frequency and duration of samples of an instru- 
ment given its input signal characteristics and the required pre- 
cision of readout. This model predicted quite well the average 
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behavior of subjects in an experimental situation. In the experi- 
ments the subject's task was to monitor a panel containing four 
instruments driven by zero-mean gaussian signals of different 
bandwidths, and to indicate whenever any of the signals exceeded 
a predetermined threshold. Since the subject's task was quite 
different than the one assumed in deriving the model, it is some- 
what surprising that there was such good agreement between theory 
and experiment. Senders (Ref. 13) explains this good agreement 
by noting that the sampling frequencies would be proportional to 

the signal bandwidths only if the signal power and the magnitudes 

* 

of significant deviations were the same for all the signals and 
that such was the case In his experiments. Thus, the agreement 
which was obtained was somewhat fortuitous since it depended on a 
unique experimental condition. 

It seemed reasonably clear that the simple periodic sampling 
model would not adequately predict behavior in more complex situa- 
tions, especially since observed data gave evidence of aperiodic 
sampling behavior (Ref. 13). Taking a cue from the fact that pilots 
often are only concerned with detection of extreme readings rather 
than with signal reconstruction. Senders proposed a conditional 
sampling scheme which would result in aperiodic behavior. In this 
approach, the human monitor is considered as a channel for the 
transmission of discrete messages and not as a channel for the 
transmission of a complete time function. In this context it is 
possible to postulate several, not necessarily mutually exclusive, 
sampling strategies. Thus, Elkind (Ref. 1*0 hypothesizes a 
strategy in which a sample is taken when the probability that the 
signal exceeds a prescribed limit is greater than some subjective 
probability threshold, whereas Grignetti (Ref. 1*1) assumes that a 


Senders defines a significant deviation as an excursion of the 
signal beyond the predetermined threshold. 
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sample Is taken when the probability of exceeding the limit is a 
maximum. Senders (Ref. 14), on the other hand, suggests a sampling 
strategy based on a "Variable Nyquist Interval". 

Unfortunately, none of the above conditional sampling models 
have been tested against experimental data. Smallwood (Ref. 15), 
however, has developed a model which has been tested against the 
data of the simple experiment described above with encouraging 
results. Although similar in some of the details, Smallwood's 
model is conceptually quite different from those previously de- 
veloped. His view of the human operator is portrayed in Figure 1. 
The model is based on two underlying assumptions: 1) the human 
oDerator bases his state of information about his environment upon 
an internal model of this environment; the model is formed as a 
result of past perceptions of his environment; 2) the human opera- 
tor behaves optimally with respect to his task and his current 
state of information within his physical limitations. Of course, 
to apply the model, one must make further assumptions. First, 
one must postulate a form for the internal model which describes 
the monitor's conception of the environment he is monitoring. 
Smallwood's approach to this problem is to assume that the monitor's 
model of each instrument is a good approximation to the true situ- 
ation. Thus, in attempting to predict the results of the labora- 
tory experiment described above, the internal model is based upon 
the assumption that the instrument readings are the result of 
passing white gaussian noise through a linear time invariant filter. 
By allowing higher-order filters, Smallwood is able to account for 
the perception of signal rate as well as signal amplitude. This 
was not possible with the previous models. The next step in 
specifying Smallwood's model is to define what is meant by the 
statement that the human operator behaves optimally with respect 
to his task. Smallwood's interpretation is that the human monitor, 
interested in detecting immediate excursions of the instruments 
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FIG.l SMALLWOOD'S BLOCK DIAGRAM OF THE HUMAN OPERATOR 



beyond the threshold, switches his attention to that instrument 
for which the probability of exceeding the threshold is a maximum. 
Note that this decision criterion is quite similar to those of 
Elkind and Grignetti. 

The last aspect of Smallwood’s model which requires specifi- 
cation is the operator's sensory system. The following assumptions 
are made by Smallwood in arriving at a model for the human visual 
perception process . 

(1) A "dead time" of 0.1 seconds is required to shift 
attention between two different instruments. 

(2) Upon fixating on an instrument a record of the in- 
strument reading is stored instantaneously within 
the short-term memory. 

( 3 ) The readout time is dependent upon the precision 
of the required reading, and this precision is de- 
pendent in turn upon the proximity of the instru- 
ment reading to one of the thresholds . 

As we have noted previously, there is very little experimen- 
tal data with which to compare the predictions of the above models. 
Even in the cases where such comparisons can be made, i.e.. 

Senders and Smallwood's predictions for the sampling behavior in 
the laboratory experiments, the experimental situation was such 
that it is virtually impossible to make inferences concerning the 
general "operational" validity of these models. However, it is 
possible to point out some inherent limitations in these models. 

The first and perhaps most significant limitation is that none 
of the models have been used to take into account, in any mean- 
ingful way, correlation between various instruments. Senders 
(Ref. 14) made a preliminary attempt to include correlation be- 
tween instruments and went so far as to run laboratory experiments 
with correlated signals. However, the results from the experiment 
were far from encouraging. Smallwood's model does not appear 
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to have any fundamental limitations with regard to signal correla- 
tion. In fact, with the appropriate choice for an internal model, 
it seems fairly clear that a method for dealing with coupling is 
possible within the framework of his model, but he has, thus far, 
eschewed any attempt to include such effects. A second signifi- 
cant limitation of these models is that none of them take into 
account, explicitly, the interaction between sampling and control 
behavior. In all cases the human operator is considered as a 
monitor only; the control requirements play little if any role in 
the selection of a sampling strategy. It is true that the 
a posteriori signal analysis which is necessary to obtain the 
parameters of the conditional sampling models will include the 
effects of control. However, this is quite different from using 
the control task in an attempt to predict sampling behavior. 

Again, Smallwood's model seems least limited with regard to in- 
cluding such possibilities; but, as in the case of signal coupling, 
Smallwood has not made any attempt to do so. 

All of the models consider the human operator as a single- 
channel device capable of processing information from only one 
instrument at a time. This approach is at variance with the re- 
cent experimental data, obtained by Levison and Elkind (Ref. 16 ), 
concerning peripheral tracking. Their experiments showed that 
under certain conditions pilots could perform two-axis compensa- 
tory tracking even though the information concerning one of the 
axes was always in the peripheral visual field. While there might 
be some argument as to whether the pilot is actually processing 
both signals simultaneously, it seems fairly clear that the abil- 
ity to perform peripheral tracking will in fact affect the pilot's 
sampling behavior and should therefore be accounted for in a 
sampling model. Still another limitation of the models is that 
no risk or cost structure has been incorporated. Equal costs are 
assigned to all instruments. Moreover, there is no cost assigned 
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to taking a sample or to switching attention. It is possible 
within the context of the Senders-type models to implicitly allow 
different costs by assigning different allowable errors in read- 
out. However, such an approach is qualitative at best. Smallwood 
assumes a dead time for shifting of attention between two different 
instruments in his model; this dead time may be interpreted as a 
cost for switching attention. Finally, we note that the Senders, 
Elkind and Grignetti models all assume that the instruments are 
fed with zero-mean gaussian signals. 

Carbonell (Ref. 17) has attempted to overcome some of the 
above limitations by developing a model of visual sampling which 
has its roots in queuing theory. If the human is assumed to be 
a single-channel processor, then one is led to the notion 
that the various information sources, i.e., instruments, queue-up 
and wait their turn to be processed. The analysis of sampling 
can then be approached as a problem in queuing theory and one can 
arrive at estimates of the probability distribution of simultaneous 
demands, the probability distribution of waiting times and estimates 
of the probability that events of interest will be missed. To be 
more explicit, Carbonell assumes the following: 

(1) Each time the observer looks at one instrument, he 
is postponing the observation of others. 

(2) The observer makes an intelligent decision before 
looking at an instrument each time, i.e., he tries 
to minimize the risk involved in not observing the 
other instruments. 

(3) This risk is represented by the probability that 
the readings may, while not being observed, exceed 
a certain threshold leading to some catastrophic 
result (such as negative altitude or lack of fuel). 
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(M) The time involved in reading each instrument 
will be assumed constant (of the order of 1/3 
of a second) for all instruments. If the observer 
looks at one instrument for a longer time, it is 
considered as a second (third, etc.) consecutive 
reading of the same instrument. In other words, 
the observer chooses this instrument again to mini- 
mize his total cost. 

(5) The observer's task in visually sampling his in- 
struments is part of a feedback loop closed through 
his control actions. 

As can be seen from his assumptions, Carbonell has indeed 
addressed himself to some of the limitations of the previous 
models. In particular he includes a cost structure and the pos- 
sibility of control actions; in regard to the latter, it is only 

fair to point out that he only accounts for control in the most 

* 

rudimentary way. Although not apparent from the assumptions, 
Carbonell also removes the restriction of zero-mean gaussian sig- 
nals. Instead, he assumes that, in the case of no control, the 
signals are generated by a sort of random walk process with equal 
probability of being above or below the value last read by Incre- 
ments that are gaussianly distributed. He assumes further that it 
is only through control action that the process can be brought to 
equilibrium, that is, that the long-term signal will be zero mean. 
However, even in such a case, the overall result is not a zero 
mean gaussian signal. Having made these assumptions, it still 
remains necessary to specify a queue discipline in order to apply 
Carbonell' s model. This is an extremely difficult task in view 
of the highly nonlinear decision process Involved. In order to 
overcome this difficulty, Carbonell decided to simulate the prob- 
lem on a digital computer and he has obtained some interesting 


He assumes that if control is effected that it is of the form 
exp(-kt) . 
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results. In particular, the simulation revealed that his model 
resulted in periodic sampling only when there was a set of low 
variance instruments and a relaxed task. Larger variances and, 
hence, harder tasks resulted in progressively more aperiodic 
sampling. This result agrees well with the observed sampling 
behavior of human pilots in corresponding circumstances. 

Carbonell's model is clearly more general and more flexible 
than any of the other models previously developed. Nevertheless 
it has not removed all the previously cited limitations. Peri- 
pheral processing is not accounted for; indeed, the concept of 
the human operator being a single-channel processor is central 
to the idea of the instrument queue. Also, coupling among instru- 
ments is not presently included in his model, although he claims 
that such coupling could be incorporated. Finally, it should be 
pointed out that Carbonell pays a heavy price in analytic com- 
plexity for the flexibility that he has obtained. It appears 
that only through extensive simulation can one obtain the pertin- 
ent model parameters and predict human sampling behavior. 

We close our review of visual sampling models by noting that 
the models of Senders (periodic sampler), Smallwood and Carbonell 
are currently being tested for their adequacy to predict behavior 
in a more complex experimental situation (Ref. 18). Pilot eye- 
movements were recorded during maneuvers "flown" on a Link CUB 
Trainer. Signals driving the instruments were also recorded so 
that they could be analyzed in conjunction with the eye-movement 
data. Unfortunately, the results of the analysis are not yet 
complete and, hence, no conclusions can be drawn at the present 
time . 
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HUMAN OPERATOR MODEL 


In this chapter we discuss the structural form of our model 
of the human operator as a control and information-processing sys- 
tem. Since the problems of interest are generally multivariable, 
state-space techniques and modern control theory are most suited 
to their study and, hence, will play a central role in the de- 
velopment. The basic assumption underlying our approach is that 
the well-motivated, well- trained human operator behaves in a near- 
optimal manner, subject to his inherent limitations and con- 
straints. This is not a new idea. Several researchers have at- 
tempted to apply optimization theory to the development of models 

of human control behavior. The most recent attempt was made by 
* 

Elkind, et al (Ref. 19). They used modern optimization theory 
to predict human behavior in a multivariable tracking task, cor- 
responding to a V/STOL vehicle in hover. The results of that 
study were quite promising and our model of controller behavior 
represents a further development of their approach. As our dis- 
cussion of the previous chapter shows, various optimization tech- 
niques have also been applied to the problem of visual sampling. 
Indeed, the models of Senders, Smallwood and Carbonell are all 
based on some optimization concept. However, as we shall see, 
our approach to the optimization of visual sampling shall be quite 
different from the ones taken by those investigators. Moreover, 
we shall be concerned with the combined control and information 
processing problem. 

The human operator model which was adopted in this study is 
shown in Figure 2 (p. 21). The model includes some of the human's 


A brief summary of other attempts to predict human controller 
characteristics via optimization techniques is given in Ref. 19. 


17 



physical limitations as well as representations for his control 
and information processing behavior. Its basic structure has 
much in common with Smallwood's model of Figure 1. The displayed 
state is essentially a representation of the human operator's 
environmental input. The block labeled perceptual processor is 
related to Smallwood's sensory system and, as we shall see, the 
data reconstructor is comparable to Smallwood's internal model. 

We now clarify some of these ideas by briefly describing 
each element in the pilot-vehicle display model of Figure 2. 

Vehicle Dynamics and Display — The human operator's basic task 
is to control, in some prescribed way, a dynamic vehicle. Several 
vehicle outputs may be of concern and the operator has several 
inputs through which he can exert control on the vehicle. It is 
assumed that the only means available for monitoring the vehicle's 
performance is a display panel consisting of several instruments. 
No external visual or kinesthetic cues are available. 

We shall assume that the vehicle dynamics are adequately 
represented by the linear equations of motion 

x(t) = A x(t) + B u(t) + w(t) (1) 

where x is the vehicle state, u the pilot's control input, and 
where w represents the external disturbances (e.g., wind gusts). 
For analytic tractability , w(t) will be considered a zero-mean, 
gaussian white noise with covariance W. 

We further assume that displayed variables are linear combi- 
nations of the vehicle states. Thus, we account for the fact 
that all of the vehicle states may not be explicitly displayed. 

As an interesting sidelight, we note that such a representation 
may allow us to investigate such things as integrated displays or 
the presentation of quickened information. 
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Perceptual Processor — The human operator has certain physical 

limitations which prevent him from making a perfect reading, ex- 

* 

cept in the simplest of cases, of the value presented on an in- 
strument. In the case of many instruments, it is clear that he 
can not make simultaneous, perfect observations. Moreover, even 
if he were able to make such observations, his limitations prevent 
him from taking instantaneous and precise action. These limita- 
tions must be accounted for in any reasonable mathematical model 
of the human operator. We now briefly discuss these limitations 
and our method for including them in the model. 

The inherent limitations of the human operator are manifold 
and complex. However, as a first approximation, the number of 
limitations to be considered can be reduced by combining phenomena 
which have similar effects. For example, the time delays associated 
with visual, central processing and neuro-motor pathways can be 
combined and represented by an equivalent time-delay which is as- 
sociated with perception. In addition, neuro-muscular dynamics 
can be approximated by a delay (Ref. 20) which can also be com- 
bined with the neuro-delays and treated as an equivalent percep- 
tual delay. Similarly, since preliminary analysis has shown that 
it will be very difficult to distinguish experimentally between 
observation noise, motor noise, and certain types of time varia- 
tions in control strategy, we can represent all three factors as 
a single equivalent "observational noise" process. The human 
operator's ability to make precise measurements is, however, a 
function of many factors. The visual environment (ambient light 
levels, glare, etc.) affects his ability to measure variables. 


For example, instruments of an on-off nature. 

Indeed, we can include "measurement noise" associated with the 
vehicle's sensors and with the displays themselves in the "ob- 
servation noise." 
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of more importance to this study, the ability to measure precisely 
a quantity would depend on the type and form of the display and on 
the location of the display in the visual field, that is, on 
whether the display is being viewed foveally or peripherally. 

With the above comments in mind, we assume that the perceptual 
process can be modelled by an equation of the form 

£(t) = ^j(t-T) + v w (t) = C x(t-T) + v^Ct) (2) 

where y(t) is the "observation" processed by the human operator, 
^j(s) is the displayed state, t is a delay and v^(t) is "observa- 
tional noise". (Note that ^(s) = C x(s), as described previously.) 

We shall assume that the observational noise v is a gaussian 

— OJ 

white noise process with a covariance that depends on the samp- 
ling strategy a>. The white noise assumption is, to an extent, 
physically unreasonable but it is a good approximation to an ob- 
servational noise process that is wideband with respect to the 
system bandwidth. Numerical values for the covariance will depend 
on where the pilot is looking (i.e., whether he is viewing an in- 
strument peripherally or foveally) and on relevant features of the 
display panel. Precise quantitative determination of these values 
is, of course, an extremely difficult task and it is not presently 
clear exactly how they will be measured. However, as we shall see, 
in some instances a coarse approximation to the values of may 

be sufficient. The method we have used for representing the per- 
ceptual process makes the choice of a suitably descriptive class 
of allowable sampling strategies an important aspect of the prob- 
lem. However, before turning our attention to the determination 
of sampling behavior, let us examine the control and data recon- 
struction portions of our model. 
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FIG. 2 MODEL STRUCTURE 
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Controller and Data Reconstructor — In this section we give 
precise meaning to the human operator's control objectives and 
develop a model for his resulting control characteristics. We 
also postulate a model for describing the manner in which the 
operator reconstructs the data necessary for control from the in- 
formation obtained by sampling the instruments on the display panel. 

We assume that the pilot's control objective is to select an 
input u(t) to the dynamical system (given by Eq . (1)) so as to 
minimize a cost functional of the form 


J (w,u) 


| t f'~“ ^f 1" - + - - + ^ (w,t)]dt|(3) 


The quantity of(w,t) in the cost functional is included to allow 
a "direct" cost for sampling. It does not explicitly depend on 
■the control input u and thus, assuming a fixed sampling strategy 
to, the human controller 1 s characteristics are determined by the 
solution of an "optimal regulator problem" (Ref. 21 ). Clearly, 
not all real control problems can be represented in this way but 
a great many can be. Moreover, using linear dynamics and speci- 
fying a control task of minimizing the cost functional of Eq. (3) 
represents a natural extension of the classical manual control 
compensatory tracking experiments. In addition, the resulting 
analytic simplification is so great as to justify extensive con- 
sideration of problems of this type. The solution of this opti- 
mization problem is well known (Ref. 22 ) and is characterized by 
the linear feedback law 

u#(oj) = -L* x^(t) (i») 
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where x^(t) is the least mean square estimate of the state x(t), 
given the observed data £ t. The feedback gains L* are 

independent of w and depend on the weighting matrices Q and R. 

The minimum cost is denoted by 

J*(w) = min J(uj,u) = J(w,u*) (5) 

u 


The estimate x^ is obtained by "reconstructing" the system 
state from the observed data In accord with our optimization 

hypothesis, we shall assume that the data reconstructor consists 
of a Kalman filter (Ref. 22) and a least-mean squared predictor 
(Ref. 19) in cascade as shown in Figure 3. The estimator is used 
to model the human's deduction of vehicle states from displayed 
information while the predictor models the human's compensation 
for his inherent time delay. The parameters, but not the struc- 
tural form, of the Kalman filter depend upon the noise covariance 
matrices W and and hence upon the sampling strategy. On the 

other hand, it can be shown that the predictor is independent of 
a) and so can be determined a priori in conjunction with the opti- 
mal control gains L* . It is interesting to note that the Kalman 
filter requires for its implementation a model of the dynamic sys- 
tem or, put another way, a model of the environment. The connec- 
tion between our data reconstruction model and Smallwood's internal 
model of the environment is then clear. 


1 

It is well known that in the absence of time delay the Kalman 
estimator provides the best mean squared estimate of the state 
for the conditions of this problem. Indeed, it provides the best 
linear estimate under much broader conditions. Moreover, with no 
sampling and no observation noise, the predictor provides the best 
mean squared estimate of the state when there is a delay (Ref. 19). 
We have assumed that with sampling, time-delay , and observation 
noise of the type described, the estimation and prediction pro- 
cesses can be separated and further, that the tandem combination 
provides for a given sampling strategy the best mean squared es- 
timate of the instantaneous state. It may be possible to prove 
this rigorously but we have yet to do so. At the least, such a 
structure should yield a good "sub-optimal" estimator. 
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Sampler — The one remaining element in our pilot model is the 
sampler. It should be absolutely clear at the outset that we are 
referring to instrument sampling and not signal sampling. By a 
sampling strategy we mean a method of deciding which instrument to 
look at directly (foveally) at different time instances. All sig- 
nals are continuously processed, but, in general, the signal on 
the "sampled" instrument has a lower noise level associated with 
it's reading. 

The determination of the human operator's sampling strategy 
will, as with other portions of our model, rest on the assumption 
that the operator behaves in an approximately optimal fashion and 
samples his instruments accordingly (for example, to minimize es- 
timation error). In other words, the human chooses a sampling 
strategy to* which minimizes J*(to) over the class of admissable 
strategies, i.e.. 


co* = arg min J*(co) (6) 

co eft 

It is important to note that <o* will, in general, depend on the 
control requirements. This will become clear in the next chapter 
where a precise formulation of the optimal sampling problem is 
given. 

In order to simplify the mathematical problems associated 
with determining to* in the above manner, it will be assumed that 
the sampling strategy is independent of the time delay and, there- 
fore, we can solve for to* by considering t*=0. Heuristically , 
this approach seems reasonable. First of all, in the situation 
of pure instrument monitoring, the effect of the time delay upon 
sampling behavior is minimal at best and a sampling model can be 
constructed which ignores this delay (as in Senders, Elkind, etc.). 
Secondly, the prediction process associated with the time delay 
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is independent of ui and depends only on the system parameters A, 

B, L*. If the converse were true then, indeed, we need only con- 
sider the case t=0. While this conclusion must be verified mathe- 
matically, it should be possible to at least show that the effect 
of time delay on sampling behavior is of secondary importance and 
may be neglected. 

Summary — A model structure for representing the human opera- 
tor in a multivariable control task has been postulated. The 
model contains elements for describing the operator's inherent 
limitations as well as his instrument monitoring, data reconstruc- 
tion and control behavior. The inherent limitations of the human 
operator have been modelled by lumping them into an equivalent 
perceptual delay and an equivalent observational noise. The noise 
covariance depends on "where" the operator looks and hence influ- 
ences the sampling strategy. The remainder of the model rests on 
the assumption that the human operator behaves optimally with re- 
spect to performing his various tasks. His control behavior is 
assumed to be that of an ideal feedback controller with gains se- 
lected so as to minimize a given quadratic cost functional. The 
control gains are independent of the data reconstruction and in- 
strument sampling processes, although the control input itself 
does depend on these processes. The data reconstruction process 
is modelled by a Kalman filter and a least-mean square predictor 
in cascade. • The structure of the data reconstructor is fixed but 
the parameters depend upon the operator's sampling behavior. 

The operator's sampling strategy is determined by solving an 
optimization problem also; in particular, by minimizing (for the 
case t=0) the functional J*(ui)(see Eq. (5)). The strategy thus 
obtained allows specification of the unknown parameters in the 
data reconstruction sub-model, thereby completing the overall 
model of the pilot -vehicle-display system. 
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Although the details of the sampling model have yet to be 
specified, several noteworthy points are already apparent. First, 
the sampling model is such that the human's monitoring behavior 
depends upon the control requirements and the control actions in 
an explicit way. Second, the ability to process information from 
the peripheral visual field is also included. Finally, there are 
no apparent restrictions with regard to coupling of signals on 
various instruments. 
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OPTIMAL SAMPLING PROBLEM 


In the last chapter we developed the structure for a model 
of the human operator in a multivariable control and monitoring 
task. We saw that, given the optimization assumption, we would 
have to solve an optimal control problem in order to predict the 
human operator's characteristics. Here, we shall make the opti- 
mization problem more precise and discuss some of its implications. 

The optimization problem itself is two-fold in nature. Con- 
trol is available over not only the plant but also over the manner 
in which information concerning the system state is obtained and 
processed. Problems of this type are embedded in a more general 
class in which one is faced with an auxiliary optimization task 
within an optimal regulator control framework. The often studied 
problem of choosing sampling times and control input which mini- 
mize a quadratic cost for a linear discrete system (Refs. 23,24) 
is an example (although not applicable to the present study) of 
this class. The design of optimal modulating signals for optimal 

linear estimation, studied by Athans and Schweppe (Ref. 25), is 

* 

also representative of this general class of problems. 

In the present study of sampling behavior we have formulated 
a problem in which the auxiliary optimization task relates to the 
measurement of the system state. Research pertinent to this prob- 
lem, for the case of discrete linear regulator systems with finite 
terminal time, has been conducted by Kushner (Ref. 27) and by 
Meier, et al (Ref. 26). Kushner considered the optimum timing of 
state observations, when the number of possible measurements is 
limited. Meier, et al, considered the general problem of "con- 
trolling" a measurement subsystem. (Such problems are referred 


It will be shown that the problems of finding optimal modulations 
and finding optimal sampling strategies have several aspects in 
common. 
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to as "measurement-adaptive" problems, and include among others, 

the optimum timing problem and the problem of finding the optimum 

sequencing of information over a time-shared communication channel.) 

A significant aspect of their formulation is the inclusion of an 

additional term in the cost functional, representing the cost of 

* 

making measurements. They have shown that the optimal choice of 
a measurement control can be determined, a priori, by solving a 
nonlinear matrix optimization problem of the type considered 
earlier by Athans and Schweppe. 

The work of Meier, et al is most closely related to our 
method for determining human sampling behavior, and we borrow 
heavily from their ideas. We begin by extending their development, 
which was for discrete problems, to the continuous time case. 

Mathematical Formulation — In this section we shall rely 
heavily on combined optimization and estimation theory for con- 
tinuous systems (Fef. 22) in order to extend the results of 
Meier, et al (Ref. 26) to the continuous time case. We shall first 
consider finite terminal time and then extend the results to the 
infinite time case. In this manner we shall reduce the sampling 
problem to a deterministic nonlinear (matrix) optimization problem. 

In order to clarify the ensuing developments we shall elabo- 
rate further on the nature of the class of sampling strategies ft . 

In particular, we shall characterize sampling behavior as a func- 
tion of time by making ft a function space whose elements w carry 

_ * /v 

the time interval [0,°°) into a given set ft. The set ft is the 
range space for functions io(*)eft. It is the set of values assumed 


With the introduction of such a term it becomes possible to con- 
sider for example, constraints on scanning behavior which arise 
from human physiological and/or psychological factors . 
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Symbolically this is written as 


* 

by w(t) for teCO,* 0 ). 

u)(*)eft: [0 


or. 


w(t)eft for all te[0,°») . 


In this manner, choosing <i)(*)eft specifies the sampling behavior 
as a function of time and, in turn, the time variation of the 
noise covariance matrix = V(uj(t)). 

We shall now derive an explicit expression for J*(to), (see 
Eq . (5)). Letw(*)efi be a fixed sampling strategy, let t^ be 
finite and, without loss of generality, let the matrix R in ex- 
pression (3) be the identity matrix. We then define 



t 

'f 

J(u>,u;t f ) = E 

1 

fc f 

# 

[x'Q x + u'u + £ (w(t))]dt 


o 


For the linear system 


d 

dP 


x(t) 


A x(t)+B u(t)+w(t); cov[w] = W;x(0) = (D 1 


£(t) = C x(t)+v w (t); cov[ v] = V( oj( t ) ) 


( 8 ) 


r /n 

For example, if there are 3 displays one may choose = (1,2,3) • 

In this case the value of io(t) could correspond to the instrument 
being viewed foveally at time t. The set Q consists of piecewise- 
constant time functions on [0,°°). 
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the control input u*(t;t f ) which minimizes (7) is given by 

u*(t;t f ) = -L*(t)x w (t) = -B'K(t;t f )x uj (t) (9) 

where K(t;t f ) is the solution of the Riccati equation 

K(t) = -K( t ) A - A'K(t) - Q + K(t)B B’K(t) (10) 

satisfying K(t f ;t f ) = 0 . The term x w (t) is the least mean square 
estimate of x(t) and is generated by the dynamical system (the 
Kalman filter): 

~ x(t) = [A - J^(t)C'V _1 (io(t))C]£(t) - B u*(t;t f ) 

+ E aJ (t)C’V" 1 (co(t))y(t) (11) 

where the matrix E (t) is the "error" covariance matrix 

— CO 

E^(t) = E |[x ( t ) - x aj (t)][x(t) - x w (t)]'| (12) 

and satisfies the equation 


E(t) = A E(t) + E(t)A' + W - E( t )£’ ’ V" 1 CE( t ) (13) 


with £(0) =0 . 

Finally, the minimum cost 

J*(aj;t f ) = J(io,u£;t f ) (1*0 
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is given by 




t -P 

t-Ji 


tr[l t 0 (t)C , V” 1 (a)(t))CS 


(t) 


or, as an alternative expression. 


K(t;t f )+Q£ oj (t)] + X. (to,t)|dt 

(15) 


J*(t*);t f ) 



ta 


,t)}dt(l6) 


In Eq. (16) the first term represents the cost resulting from the 
driving noise w(t) and is independent of ui. The second term rep- 
resents the degradation in cost arising from the estimation noise 
V . The third term represents the "sampling cost" as discussed 

—■“tO 

earlier. 
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In the above development It is noteworthy that : 

1) L*(t) is independent of the noise processes, and 
therefore independent of w. (This is the well- 
known separability result for linear systems.) 

However, the converse Is not true as to depends 
on L* through the second term of Eq . (16). 

2) _I w (t) is independent of the terminal time t f and 
independent of L*(t). 

We now extend the above results to the case t^.-* 00 . We assume 
(although it may be possible to show rigorously) that 


lim imin J(io;u;t f )\ = min J(to;u,®) = J*(w) (17) 

t f -»* l u ‘ u 
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and that this minimum is achieved at 


u*(w) = lim u*(io;t„) 
t f -*» 1 

The left-hand limit of Eq . (17) is readily shown to exist. 


lim K(t;t^.) =? K = constant 


t f~ 


where K is the unique positive definite solution of 


0 = K A + A'K + Q - K B B'K 


we have 


lim L* ( t ) = L* = -B’K 
t f - 


Furthermore, since E^(t) is independent of t f we obtain 

lim J*(oj;t -) = tr[W K] + tr[L*E„„ (u>)L*'] + X 

I — — avg — c 


t f -x» 


■avg 


where 


= J*(u) 


W u) 




lim i- 
t f ->°° t f { 


2(<o,t)dt 


lim i- 

t f -*» z r 


^(to, t )dt 


( 18 ) 

Since 

(19) 

( 20 ) 
( 21 ) 

(to) 

( 22 ) 
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are the time-average values of £(w,t) and (ai,t) over the infinite 
interval [0,"). 

In addition, the limit (18) becomes 

u*(w) = -L* x^t) ( 2 3) 

where x (t) is generated by Eq. (11) with u*(t;t^.) replaced by 

-k* 

The element a)* ( • ) £ft (or equivalently a)*(t)eft for te[0,°°)) 
which minimizes J*(w) will be used to predict the pilot's scanning 
behavior. This is in accordance with our assumption that the 
pilot acts in a (near) optimal fashion and thereby chooses an 
optimal sampling pattern. It remains, therefore, to minimize 
J*(u>) over Q , or equivalently, since the first term in J*(w) is 
independent of a), to minimize 

Kw) = tr[L*E avg (o>)L*'] + J? avg U) < 2l O 

The mathematical optimization problem of finding a)* is ex- 
ceedingly difficult at this stage because of the form of the de- 
pendence of 1(a)) upon a). For example, if a)^(*) and a^t*) are two 
elements of ft which differ only over a finite interval [a,b]c[0,~) 
then 


I(a> 1 ) = I(w 2 ) 

This implies that there is no unique element a)*(*)eft which mini- 
mizes 1(a)).* Thus, the application of variational techniques to 

^ ■ ■ — ■ ■ " 

In fact there are an infinity of elements in ft which minimize 1(a)). 
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solve the problem In Its present form is virtually useless, and 
it becomes necessary to introduce further restrictions on the 
class ft . 

The assumption we shall make is that any element to(*)eft is 
periodic with period T < °°, where T is fixed but arbitrary. Hence, 

oj(t) = io(t+T) for all te[0,°°) (25) 

The mathematical assumption (25) is tantamount to the physical 
assumption that the pilot's sampling behavior is periodic with 
period T. 

With the assumption (25) we have, in turn, that V(to(t)) is 
also periodic with period T. It is then possible to show: 

1) There exists a unique periodic solution of the 
variance equation (13) having period T. Call 
this solution J£(oo,t). 

2) For any positive semi-definite matrix F let 
£p(u),t) be the solution of Eq . (13) satisfying 

£p,(u),0) = F, then 

lim !„(<*>, t) = 1(a), t) ( 26 ) 

t-*» * 


The above two statements imply that 


I(oj) = | tr 


T 

L* | T(oj,t)dt L*' 
o 


T 

+ ijr | £(w,t)dt 

o 


(27) 


Consequently, the determination of to* has been reduced to the 
following nonlinear optimization problem: 
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Optimal Sampling Problem : Given T < ». Find the element 

which minimizes I(w)> where T(u),t) satisfies 

jr E(t) = A'Z(t) + E(t)A + W - £( t )C ' V _1 ( w( t ) )C_E( t ) ( 28 ) 


with the boundary condition 


Z(0) = £(T) > 0 . 


(29) 


Thus far, the sampling period T has been considered a fixed but 
arbitrary parameter. However, in order to completely specify the 
sampling behavior of the human operator, we must also specify T. 
This is accomplished, in accordance with our optimization hypothe- 
sis, in the following manner. For a particular value of T the 
optimal sampling problem can, in principle, be solved for id* and 
I(u>*) . Thus, (*)*(•) will parametrically depend on T and we write 


w»( • ) = w*( • ) 
I(io*) = I(u>») 


(30) 


We now vary T, and for each T compute u)^,( • ) as well as I(co^). 
Thus, I (o)^,)» will be a scalar function of T and we let 

T* = arg min 1(a)*) (31) 

T 1 

be the requisite choice of scan period (i.e., the optimal scan 
period). In this manner, w*# will describe the optimal sampling 
strategy . 
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In order to accomplish the minimization indicated in Eq . (31), 
it will be necessary to introduce certain nonrepetitive assumptions 
and the class ft. As T increases, the set ft will encompass a wider 
class of sampling strategies and not merely repetitive cycles of 
those sampling strategies contained in a set ft for smaller values 
of T. (Recall that a function which is periodic in T is also peri- 
odic in 2T. ) 

Solving the Optimal Sampling Problem — The deterministic opti- 
mization problem derived above is of a special form. The error 
covariance matrix Z(t) may be interpreted as a system state, the 
matrix Riccati equation plays the role of a dynamic system and the 
function io(t) is regarded as a control input. Such a nonlinear 
optimization problem will normally have to be solved computation- 
ally. Several approaches, based on different characterizations 
of the optimal solution, are possible. 

Athans and Schweppe (Ref. 25) used a matrix version of the 
Maximum Principle in their investigation of a nonlinear optimiza- 
tion problem of this type. They considered the problem of mini- 
mizing 


I(o>) = tr Z_(T) 

subject to the boundary conditions £(0) = = given; the control, 

to(t), was constrained. Using the (matrix) Maximum Principle, they 
obtained necessary conditions for optimality. Such conditions can 
serve as the basis for various computing algorithms. However, 
since our cost functional is of a different form and since the con- 
straints on to will not be local in nature if we constrain the 
sampling rates, the techniques of Ref. 25 would have to be extended 
before they could be applied to the problem of determining to* . In 
addition to using the Maximum Principle to compute to*, there also 
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exists the possibility of applying it (for the free terminal time 
case) to the problem of determining the optimal sampling period. 

Meier, et al (Ref. 26) used dynamic programming to solve some 
measurement adaptive problems. Dynamic programming methods have 
distinct advantages so long as one is not confronted with the 
"curse of dimensionality" (Ref. 28). For the optimal sampling 
problem, the dimensionality, from a dynamic programming stand 
point, is determined by the number of instruments to be sampled. 
Thus, in problems in which the number of instruments to be sampled 
is not too large, dynamic programming techniques might prove quite 
useful . 

Although the development of theoretically based computational 
algorithms for solving the optimal sampling problem is a desirable, 
if not necessary, goal, it is possible to solve simple versions of 
the problem by numerical search techniques. It is always possible 
to calculate the cost I(w^) for a given . This calculation is 

nontrivial because of the split boundary conditions ( 29 ) on I/t). 

In order to evaluate I(w^) it is first necessary to compute I/O) 
explicitly. Once this is accomplished, _I(t) is readily obtained 
by integrating Eq . (28). We note that since Eq . (28) possesses a 
unique periodic solution (corresponding to a given periodic 
sampling strategy) it is possible to compute I/O) by means of the 
convergent algorithm 

I( i+ 1) (0) = CI 21 (T,0) + l22 (T > 0) Ii (0)][ Iii (T > 0) + I^C^OI^O)] -1 

(32) 
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where the 2n x 2n matrix 


I(t,t 0 ) 


^ll(t,t 0 ) 

-12^ t,t 0‘ ) 

I 2 i(t>v 

I 22 (t > t o ) 


is the transition matrix associated with 


Z(t) 


-A' 1 -C V~ 1 ( to ( t ) )C 
-W J A 


( 33 ) 


(34) 


i . e . , 

I(t,t 0 ) = Z(t)t(t,t 0 ) ; I(t 0 ,t 0 ) = I (35) 

Thus, I(w) can be evaluated for a given m and one can numerically 
compare the costs 1 ( 0 )^) and I(w 2 ) associated with two different 
sampling strategies and u> 2 . In this manner it becomes possible 
to perform a numerical search for w* for a given value of T. One 
can then repeat the procedure for various values of T and thus 
determine the minimum of I with respect to co and T. 

Summary and Comments — In this chapter a precise formulation 
of the optimum sampling problem was developed. In the course of 
this development several points worthy of further elaboration 
emerged. First, and perhaps most noteworthy, was the assumption 
that the sampling was periodic. This assumption is not as restric- 
tive as it may first seem. Within a period T, any arbitrary se- 
quence of instrument selections is possible, provided that the 
entire sequence is not repeated. This means that each instrument 
may be looked at several times during a single period T and that 
the interval between these looks is not necessarily the same. 
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The assumption of periodicity is necessitated by the consideration 
of an infinite time problem and is tantamount to assuming that 
there will be some "steady state" sampling behavior. If we so 
choose, the optimization framework can be used to predict human 
sampling behavior for a control task of finite duration and thus 
obviate the need for the periodicity assumption. This is accom- 
plished by minimizing J(u(*),w) for a finite t f (as opposed to 
taking the limit as t f -» ») . The results of such an approach will 
be very similar to those obtained for the infinite time case, ex- 
cept that t^ replaces T, and the initial condition £(0) must be 
given in the problem statement. For this case the optimal co* will 
represent the scan pattern for the run duration and not averaged 
scanning behavior. It should be noted that If t f is finite the 
optimal control gains L* are no longer constant. 

A second point is the possibility of introducing sampling 
rate constraints in one of several different ways. Such con- 
straints are clearly present in the human operator and, if we are 
to obtain meaningful results from our model, it seems likely that 
we will have to account for them in our representation. Explicit 
sampling rate constrains can be considered by bounding the derivative 
of u>(t); this artifice is similar to that suggested by Athans and 
Schweppe (Ref. 25) for constraining signal bandwidths . Alterna- 
tively, it may be possible to include a cost for high sampling 
rates in the* functional I(w) by choice of a suitable form for the 
term «£(u>,t). Sampling rates can also be constrained implicitly 
by introducing certain transition or "dead" times which corre- 
sponds to physiological read-out or data acquisition times. This 
was the approach taken by Smallwood and is also the approach we 
shall use in the numerical example of the next chapter. 

The connection between control requirements and sampling 
behavior was also clarified in the above development. For the 
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joint control and sampling problem the cost function I(w) is 
given by (see Eq . (2*0) 


!(“) = ’ ] + ^avg (w) 


Now, if the operator's task is simply to monitor the instruments 
and obtain a minimum variance estimate of the system state, it 
can readily be shown that the optimal sampling strategy is ob- 
tained by solving for w* to minimize 


I (w) = 


tr[ W 


( oa ) 
avg v 


Such a cost function would assign equal weight to errors in esti- 
mation in any of the state variables. If the operator were in- 
structed to consider some state variable errors more important 
than others, he could do so by appropriately modifying the cost 
function, that is, by choosing I to be of the form 


= tr[M 1 M ' ] 

avg- 


£ (w) 

avg' 


The matrix M weights the various estimation errors according to 
the preselected criteria. But, this is exactly the role played 
by the optimal gain matrix L* in our formulation of the joint 
sampling and control problem. Thus, the sampling strategy oj is 
selected to minimize a weighted estimation error where the weight- 
ing is consistent with the contributions of the state variables 
to the optimal feedback law. The matrix L* depends only on the 
original weighting matrices, Q and R, in the "overall" performance 
index. Hence, it is possible to investigate, in a systematic way, 
the changes in human sampling behavior which result from differ- 
ent control requirements (that is, from different weightings Q 
and R) . 
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Finally, we saw that in order to determine the sampling be- 
havior, it was necessary to solve a nonlinear deterministic opti- 
mization problem. Analytic solution to the problem is highly un- 
likely and we are faced with the probability that solutions can 
only be obtained computationally. Several approaches to obtain- 
ing an efficient computational algorithm were discussed but this 
remains a prime area for further research. It is sometimes feas- 
ible, however, to obtain solutions by a numerical search and this 
is done for a simple example in the next chapter. 
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A SIMPLE EXAMPLE 


In this chapter the foregoing concepts and theory are applied 
to a specific control and monitoring task. In order to illustrate 
the method and avoid cumbersome numerical calculations, we shall 
consider a simplified example. We shall illustrate: 1) a method 

of choosing H and the dependence of V(w) upon o>, 2) a method for 

constraining sampling rates, and 3) the effects of certain para- 
metric variations upon predicted sampling behavior. 

A Two- Axis Tracking Problem — We assume that a human operator 
is presented with two separate displays each being driven by the 
output of a first order dynamical system of the form 

x^t) = a 1 x 1 (t) + b i u i (t) + w^t) ; covEw^t)] = (36) 

Thus, the two sets of dynamics (i = 1,2) are completely noninter- 
acting. We assume that x^ and x 2 are presented on display 1 and 
2 respectively, and that the quantities viewed by the human opera- 
tor will be (cf. Eq-. (8)) 

y i ( t ) = x^t) + v^oj) ; cov[v 1 («)] = v^Cw), i = 1,2 (37) 

The additive observation noise v. (w) is associated with display i 

1 * 

and depends on the sampling behavior io(t). We defer, for the 
moment, a discussion of this dependence and of the sampling class 
fl. 

The control task is to pick u 1 (t) and u 2 (t) so as to minimize 


It Is assumed that v 1 (t) and v 2 (t) are independent. 
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V 

J(u 1 ,u 2 ) = lim E j [q 11 x^(t) + q 22 x 2 (t) 
t f" W I f o 

+ u^(t) + u 2 (t)]dt ■ (38) 

Our objective is to predict the human sampling behavior associated 
with the given control task. We shall accomplish this by solving 
the optimal sampling problem for w*. 

Choice of Sampling Parameters — The solution of the optimal 
sampling problem requires the explicit specification of the class 
of sampling strategies ft and the functional relationship between 
V(w) and w(t). For the example being considered we shall include 
sampling rate constraints within these specifications. 

1. Sampling Class 

We consider a fixed (but arbitrary) scan period, T. At 
any time tE[0,T] it is assumed that the pilot is either looking 
foveally at display 1 or display 2, or else, is in the transi- 
tional phase (i.e., switching attention). We let u(t) = 1, 2 or 
0 to correspond to each of the above cases, respectively. Hence, 

A 

the set ft which is the range space of aj(*)eft (see pg. ( 29 )) is 

ft = (1,2,0} (39) 

A 

and to(*)eft maps [0,T] into ft. 

Note that w(t) so defined is piecewise constant, and over any 
time interval may be described by a sequence of values (1,0 ,2,0,1, ...} . 
Notice further that sequences of the form {...,1,2,...} are not 
allowed. We assume that every transition of foveal attention 
between displays requires a finite transition or "dead" time, t . 
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Examination of some of our pilot eye movement data indicates that 
a reasonable value of t is 0.1 sec. (This was also the value 
Smallwood assumed.) 

We are interested in a sequence in which the pilot samples 
each instrument once per scan period. Thus, the sequences of 
values of w(t) for any i s (assuming for convenience, that 

instrument 1 is sampled first) (1,0, 2,0}. If we let tj and t 2 be 
the times that a pilot spends in viewing instrument 1 and 2, re- 
spectively, we have 

T = + t 2 + 2t Q (40) 

Thus, any element uiefi can be characterized by the single number 
t^ (or t 2 ) since T and tp are given. In summary, then, any ele- 
ment to(*)eO is piecewise constant over [0,T] and takes on the 
sequence of values {1,0, 2,0). 

2. Characterization of V ( u> ) 

Since co(t) can take on only one of three values, (0,1 
or 2) V(io) may be characterized by three matrices V(l), V(2), 

V(0). The matrices V(l) and V(2) correspond, respectively, to 
the observation noise covariances for foveal viewing of display 1 
or display 2, while V(0) is the corresponding covariance matrix 
for switching attention. Since we have assumed independent noise 
processes, the above matrices are diagonal. Hence, 

V(u>) = diag[v 11 ( w) , v 22 (to)]. 

Here, we shall assume that the elements of V(l) and V(2) are 
given quantities in the problem formulation. These parameters 
relate to, and are identified with, the display system and its 
interaction with the human operator. For Instance, the numbers 
vnd) and v 22 ( 2 ) will be less than v^(2) and v 22 (l), respectively. 
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since foveal viewing of a given display will result in less obser- 
vational noise than will peripheral viewing. Similarly, if the 
display separation is increased (or decreased) we might expect the 
peripheral noise covariances v^(2) and v 22 (l) to similarly in- 
crease (or decrease). 

The elements of V(0) will be relatively large since we assume 
that no display is seen accurately during the transition between 
instruments. An alternative assumption would be to consider that 
no instrument is viewed foveally but that some (in this case both) 
are viewed peripherally during a transition. This latter assump- 
tion would also result in relatively large values for the elements 
of V(0). Examination of the variance equation (28) shows that as 
the elements of V increase, the solution, £ becomes increasingly 
independent of the magnitudes of these elements. This implies 
that a fairly gross approximation to V(0) may be adequate for pre- 
dicting sampling behavior. If such is the case it would be for- 
tuitous since accurate determination of the elements of V(0) seems 
an extremely difficult task. Indeed, determining the required 
values of V(l) and V(2) is also nontrivial. However, for our 
present purposes, we may avoid these difficulties by making a 
priori selection for the values of the various noise covariances. 

Optimal Sampling Problem for Two-Axis Tracking — We are now in 
a position to solve the optimum sampling problem. We are given 
the system dynamics, cost criterion, class of sampling strategies 
and observation noise covariances. It is then easy to show (since 
display 1 and 2 are independent) that the optimal sampling problem 
reduces to: 

Given T < <», find the element which minimizes 

( T T ) 

I(w) = / a 11 (t)dt + * 2 / o 22 (t)dtj (41) 
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Since io(t) assumes the values 1, 2 or 0 in the sequence 

{1.0, 2,0} and since T and t are fixed, the determination of w#(*) 

o 

is equivalent to the determination of t^. Thus, I(w) is a function 
of the single parameter t^ and the value of t^ which minimizes I(w) 
can be obtained by a simple numerical search. For a fixed value 
of t^ the value of I(w) is obtained by first solving Eq . (43) for 
0 ^ 2 ( 0 ) an< ^ '^ 1 ^ s ^- s readily accomplished since v 2 i^) and 

v 22 (t) are merely piecewise constant. We then integrate Eqs . (42) 
and (41) to obtain I(to). 

Thus, for a fixed value of T we can obtain o)*(»). In order 
to obtain the optimum scan period T it is necessary to investigate 
the behavior of I(to*) as a function of T. The value of T which 
minimizes I(io*) is denoted by T* and serves as a prediction of 
human scan period. By the nature of the class n, T* must neces- 
sarily be greater than 2t Q (except for the case when foveal 
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attention is directed entirely at one display). Finally, the 
sampling function to* which corresponds to T* will serve to pre- 
dict mean dwell times for instrument 1 and 2. 

For numerical convenience it will be assumed that 
v n(0) = v 11 (2) and v 22 (0) = v 22 (l). This corresponds to peri- 
pheral viewing of both displays during a transition between in- 
struments. This assumption is not expected to appreciably affect 
sampling behavior since v i;l ( 2) and v 22 (l) will be greater than 
v n(l) or v 22 (2) (see discussion on pg. (46)). 

Results and Discussion — In this section we shall examine some 
of the two-axis sampling behavior predictions of our optimal 
sampling model. We first examine a "nominal" case and determine 
the optimal sampling period and the average dwell-times for each 
instrument. Then we examine the effects of various system changes 
on the predicted sampling behavior. We shall investigate system 
changes of the following types: 

1. Changes in controlled element dynamics. 

2. Changes in noise covariances. 

3. Changes in cost functional weightings. 

The Nominal Case — We choose for a nominal case the situation 
in which both axes are identical, i.e., the dynamics, the driving 
noise covariances, the associated observation noise matrices, and 
the cost functional weightings are the same for each axis. The 
nominal controlled element dynamics correspond to the case a 1 =a 2 =0 
(this is equivalent to the k/s dynamics, with k=l, frequently used 
in tracking experiments). The remaining parameters are chosen 
arbitrarily to be 
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A transition time t Q of ,1 seconds is assumed throughout. 

The results for the nominal case are shown as the circled 
points in Figures 4-12. We see that the optimum sampling model 
predicts, as one would expect, that an equal time is spent on each 
display. Note, however, that this is not 50 % of the sampling 
period since 2/10ths of a second is lost in the switching of at- 
tention. The optimal sampling period T* is found to be approximately 
1.1 seconds. The cost due to sampling I(<u) is approximately 2.8. 

No significance can be attached to the numerical values of T* and 
I(w) at this time since we presently have no experimental data 
corresponding to an equivalent situation. Levison and Elkind 
(Ref. 16) did look at a two-axis tracking task with k/s dynamics 
on each axis. They found that when the inputs to both axes were 
the same the operator spent about equal time viewing each axis. 

They obtained average sampling periods in the neighborhood of 2.5 
to 3 seconds. However, they used relatively low bandwidths inputs 
and, hence, there is no reasonable basis for comparing their re- 
sults with those obtained here. 

Effects of Controlled Element Dynamics — In this section we 
examine the effects of varying a^ and aj, that is, the controlled 
element dynamics, while holding the remaining parameters fixed. 

The results of such variations are shown in Figures 4-6. The re- 
sults agree well with intuition. From Figures 4a and 4b we see 
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that a larger percentage of time is spent viewing the axis which 
is more difficult to track, that is, the axis with the more un- 
stable dynamics. Indeed, as one axis becomes more stable with 
respect to the other, one soon reaches the point where the best 
thing to do is to spend all of the time looking foveally at the 
more unstable axis; the more stable axis is then tracked solely 
on the basis of information obtained from peripheral viewing of 
the associated display. 

In Figure 5 we see the effects of changes in the controlled 
element dynamics on the sampling period. Specifically, we have 
plotted the minimum sampling period T# as a function of a 2 for 
various fixed values of a^. This family of curves has several 
interesting, and perhaps important, properties. First, for a^ 
fixed, the optimal sampling period T* has a unique minimum when 
considered as a function of a^ and this minimum occurs, at least 
numerically, when a 2 is equal to a^. Moreover, the locus of these 
minima, i.e., the envelope of the family of curves, seems to be a 
well-defined, smooth curve. We hasten to point out that we pres- 
ently have no analytic results which would support drawing general 
conclusions from these findings. However, it is difficult to 
believe that the numerical results are purely coincidental. An- 
other interesting point concerning these curves is that T* has a 
sharper minimum for values of a^ corresponding to more stable 
dynamics. The value of this minimum, however, decreases as the 
systems become more unstable. The decreasing minimum sampling 
period can be explained by noting that as both systems become 
more and more unstable it becomes necessary to view each display 

foveally more often in order to avoid large estimation errors 

£ 

(associated with peripheral viewing). The sharpness of the minima 


Recall that scanning frequency is inversely proportional to T* . 
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is more difficult to explain. It may be related to the decreasing 
value of the minimal optimal sampling period; thus, as Min T* de- 
creases, that is, as the systems become more unstable, the rela- 
tive contribution of the dead time 2t Q to the sampling period in- 
creases and, consequently, small variations in the dynamics of one 
system do not appreciably alter the optimal sampling period. 

Finally, the effects of changes in controlled element dynamics 
on the cost for sampling is illustrated in Figure 6. As expected 
the costs increase as the systems become more unstable. 

Effects of Observational Noise Covariances — As noted earlier, 
the observation noise matrix will be used to account for the effects 
of imprecise measurements. This lack of precision may result from 
display related phenomena or inherent pilot limitations. Quanti- 
tative specification of the observation noise is, however, an ex- 
tremely difficult task. Here, we content ourselves with an examin- 
ation of the changes in sampling behavior which result from varia- 
tions in both foveally and peripheral noise levels, as well as in 
the ratio of the two. All other system parameters are fixed at 
their nominal levels for this investigation. 

In Figures 7 and 8 we show the effects of changes in periph- 
eral and foveal noise levels on the optimal sampling period and 
the sampling cost, respectively. Several conclusions may be drawn 
from examination of these figures. First, for fixed values of 
peripheral noise, increases in the foveal noise result in increases 
in the sampling period. This may be explained by the fact that 
as the foveal noise level approaches that of the peripheral noise, 
the quality of information obtained by foveal viewing degrades to 
the point where it is not much better than that which can be ob- 
tained by peripheral viewing. Consequently, there is less and less 
reason to switch foveal attention. In the limit, when the two 
noise levels are equal, there is no reason to switch foveal atten- 
tion and one would expect the sampling period to approach infinity. 
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The curves of Figure 7 do indeed show very rapid increases in the 
sampling period as the foveal noise level approaches the peripheral 
noise level. On the other hand, for fixed foveal noise levels we 
find that as the peripheral noise increases, the sampling period 
decreases and rapidly tends to some nearly constant value. The 
fact that for large peripheral noise levels, small changes in those 
noise levels do not appreciably affect the sampling period, is not 
surprising and was discussed earlier. As peripheral noise levels 
increase the amount of information which can be obtained from 
peripheral viewing decreases. Beyond a certain point, the only 
useful information is obtained during foveal viewing and further 
increases in peripheral noise levels have negligible effect on 
sampling behavior. 

An appealing interpretation of the above results is possible. 
The foveal noise level can be considered to be a function of the 
display design. High foveal noise levels correspond to a poor 
display. Thus, if the foveal noise is high, longer reading times 
and, consequently, longer sampling periods are required. The ratio 
of foveal noise to peripheral noise, for a fixed foveal noise level, 
can be interpreted as a measure of display separation. As the dis- 
plays are separated this ratio decreases or what is the same, for 
fixed foveal noise, the peripheral noise increases. Thus, one 
reaches a point of display separation beyond which increases in 
the separation do not appreciably affect the sampling period since 
nearly all useful information must be obtained from foveal viewing. 
Conversely, as the foveal to peripheral noise ratio increases, the 
displays are moved closer together and the necessity for visual 
sampling decreases. When the ratio is one, the instruments may be 
considered superimposed or integrated. Then, there is no neces- 
sity for visual sampling and the sampling period approaches infin- 
ity. 
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Finally, the cost I(oj) increases monotonically with increases 
in either foveal or peripheral noise. The costs also tend to level 
off as the peripheral noise increases so that the sensitivity of 
the cost to changes in peripheral noise level for large peripheral 
noise levels is small. These results are consistent with our ex- 
pectations . 

Effects of Input Noise Covariance — The effects of changes in , 
input noise levels, for fixed observation noise, on sampling be- 
havior are illustrated in Figures 9 and 10. Figure 9 shows that 
as the input noise level increases, the sampling period T* decreases. 
Increasing the input noise covariance may be interpreted as making 
the task more difficult. This, as in other instances, results in 
more frequent sampling. As expected, the effect is similar to 
that obtained when the input noise level is fixed and the observa- 
tion noise is decreased. It would appear that the significant 
parameter is the ratio of input and observation noise covariances. 

In Figure 10, we see the effects at having different input 
noise levels on each axis. In particular, we have plotted the 
percent time spent on instrument 2 vs. the driving noise level on 
axis 2 (w^ 2 ) for various values of axis 1 input noise levels (w^). 

We find that for fixed values of w^ the percent time spent on in- 
strument 2 increases as W £2 increases. This apparently results 
from the fact that the control task becomes more difficult as the 
input noise level increases. 

Effects of Control Requirements — One of the more significant 
aspects of the optimal sampling model is the interaction between 
control requirements and sampling behavior. Here, we briefly ex- 
amine the effects of changes in control requirements on sampling 
behavior for the two axis tracking problem. 
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In the optimization framework we are using, control require- 
ments are explicitly stated in terms of a selection for the weight- 
ing matrices Q and R. We have "normalized" this selection by 
letting R be the identity matrix. Thus, for the two axis tracking 
problem we need only choose and q 2 (since Q is assumed diagonal). 
The most significant quantity is actually the ratio of these two 
weighting coefficients for this simple case. 

Figures 11 and 12 show the effects of changing the ratio 
q^/qf, for fixed q^, on the fraction at time spent on instrument 
2 and the optimal sampling period, respectively. The results are 
as expected. Figure 11 shows that as q 2 decreases, i.e., as con- 
trol of x 2 becomes less important, the time spent on instrument 2 
also decreases, in spite of the fact that the two axes are iden- 
tical in all other respects. It Is interesting to note that the 
point at which instrument 2 is not looked at, at all, occurs prior 
to q^ = 0. This is apparently due to the inclusion of a cost for 
switching attention (2t ). Thus, as q 2 decreases, a point is 
reached where the "penalty" for switching attention exceeds the 
benefits of obtaining better information, by foveal viewing, con- 
cerning x 2 « 

From Figure 12, we see that the optimal sampling period T* 
is a minimum when q 2 /q 1 = i.e., when both states are equally 

weighted. In addition, small departures from equal weighting do 
not materially increase the sampling period. As q 2 /q^ becomes 
either very small or very large the sampling period increases 
rapidly. This is to be expected since both cases correspond to 
a situation in which one instrument is not very important and, 
hence, there is little need to look at it. 
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CONCLUSION 


In order to understand the effects of certain display changes 
on overall pilot-vehicle system performance, it is necessary to 
investigate the correlation between control objectives, display 
properties and various aspects of human behavior. If such an in- 
vestigation is to result in objective measures of display effec- 
tiveness, the interactions among display, pilot and vehicle must 
be described analytically. This, in turn, requires the develop- 
ment of mathematical models for the various sub-systems, including 
man himself. In this report we develop a model of the human opera- 
tor which we feel will be well suited to analytic display evalua- 
tion techniques. 

The model of the human operator in a multivariable control 
and monitoring task is developed by considering the human as an 
information processing-control "system". The key idea underlying 
the model is the assumption that the well-trained, well-motivated 
human operator behaves in a near optimal manner, subject to his 
inherent limitations. The model contains an element for describ- 
ing the human’s limitations in addition to representations of his 
instrument monitoring, data reconstruction and control behavior. 

The human's limitations are modelled by combining them into 
an equivalent perceptual time delay and an equivalent observational 
noise. The observation noise covariance depends on where the opera- 
tor "looks" and, hence, allows the effects of peripheral vision 
on tracking and instrument "sampling" behavior to be modelled. 

The operator's control behavior is assumed to be similar to 
that of an ideal feedback controller. The feedback gains, which 
multiply estimates of the system states, are selected so as to 
minimize a weighted sum of mean squared tracking errors and con- 
trol effort. 
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It is assumed that the human operator processes the informa- 
tion he obtains from scanning the display panel so as to recon- 
struct a "best" estimate of the data necessary for control. The 
"data reconstructor" consists of a Kalman estimator and a least 
mean squared predictor in tandem. The structure of the data re- 
constructor is fixed but its parameters depend on the operator’s 
instrument sampling strategy. 

Since an adequate representation of instrument monitoring, 
or visual sampling, behavior is central to the problem of display 
evaluation, we devoted most of our attention to this aspect of our 
human operator model. We reviewed existing sampling models and 
found that they had certain inherent restrictions which we felt 
would limit their applicability. Rather than attempt to modify 
these models, we postulated a new sampling model which was con- 
sistent with other portions of our model in that it was based 
upon an underlying optimization approach. In this approach the 
operator is viewed as an adaptive-measurement system; he decides 
where he will direct his foveal visual attention on the basis of 
a preselected optimality criterion. This criterion depends ex- 
plicitly on the control task or control requirements. Peripheral 
vision is accounted for and there are no restrictions with regard 
to coupling of signals on various instruments. Provisions for 
sampling cost and sampling rate constraints are also included in 
the model. 

The assumption of the optimality of the human operator leads 
to the necessity for solving a multifaceted optimization problem 
in order to predict specific operator characteristics. We have 
formulated this problem precisely and have discussed methods for 
obtaining its solution. Finally, we examined a simple two-axis 
compensatory tracking problem and used the model to predict in- 
strument sampling behavior. The results exhibit the general 
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characteristics one would expect from a human operator performing 
a similar task. 

Some Topics for Further Investigation — Thus far, we have postulated 
a model for the human operator in a controlled environment which 
has interesting and intuitively appealing properties. We have not, 
however, verified that this model is a valid representation of 
human behavior. Previous experiments (Ref. 19) have shown that 
the controller and predictor portions of the model yield reason- 
able representations of the human operator. It is now necessary 
to determine the efficacy of the sampling and estimation models. 
Hence, the next step in our research program will be to test these 
models against existing experimental data. 

In the present study the observational noise covariances were 
chosen arbitrarily. It is our hope that numerical values for 
these quantities can be measured in appropriate experiments. 

Since the noise covariances depend on the display and the environ- 
ment as well as on intrinsic human properties, their determination 
will be extremely difficult. Indeed, it may be necessary to con- 
sider the noise covariances as free parameters which are chosen 
so as to match experimental data. This would be somewhat unfor- 
tunate and it seems that considerable thought should be given to 
methods for experimentally determining the observation noise. 

In addition to the questions concerning the model which must 
be resolved experimentally, there remain problems of a theoretical 
nature. In the theoretical development we made several assump- 
tions concerning the separability of various elements in the model. 
It would be useful to show vigorously under what conditions these 
assumptions hold. Of greater interest is the examination of 
methods for mathematically describing the class of sampling strate- 
gies ft . The specification of this class has both mathematical 
and physical implications. For instance, in the example we 
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selected SI such that the range space of co(*)eft was the finite set 
SI = {0,1,2}. Although the results are interpreted in terms of 
average behavior (because of the infinite interval and the result- 
ing periodicity assumption), this choice of SI corresponds, essen- 
tially, to defining deterministic sampling strategies. Moreover, 
this choice of & introduces mathematical complexities since w does 
not take on a continuum of values. A characterization of Q which 
eliminates this type of complexity and, at the same time, allows 
for a direct probabilistic interpretation of the optimal sampling 
strategy co* is possible. This, and other, descriptions of the 
sampling class warrant investigation. 

Finally, there is the question of computing algorithms for 
solving the optimal sampling problem. It seems reasonably clear 
that the numerical search technique used in the example will prove 
much too tedious for more complex problems. The investigation 
of more efficient algorithms thus remains a prime area for research. 
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